This repository contains the official implementation for DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation.
Project Page | Arxiv accelerate.mp4 News2023.12.22: add experimental support for ImageDream, check imagedream.yaml.2023.12.14: add support for Stable-Zero123, check image_sai.yaml.2023.10.21: add support for MVDream, check text_mv.yaml.Colab demoImage-to-3D:
Text-to-3D: Gradio demoImage-to-3D: 
Run Gradio demo on Colab: Installpip install -r requirements.txt# a modified gaussian splatting (+ depth, alpha rendering)git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterizationpip install ./diff-gaussian-rasterization# simple-knnpip install ./simple-knn# nvdiffrastpip install git+https://github.com/NVlabs/nvdiffrast/# kiuikitpip install git+https://github.com/ashawkey/kiuikit# To use MVdream, also install:pip install git+https://github.com/bytedance/MVDream# To use ImageDream, also install:pip install git+https://github.com/bytedance/ImageDream/#subdirectory=extern/ImageDreamTested on:
Ubuntu 22 with torch 1.12 & CUDA 11.6 on a V100.Windows 10 with torch 2.1 & CUDA 12.1 on a 3070.UsageImage-to-3D:
### preprocess# background removal and recentering, save rgba at 256x256python process.py data/name.jpg# save at a larger resolutionpython process.py data/name.jpg --size 512# process all jpg images under a dirpython process.py data### training gaussian stage# train 500 iters (~1min) and export ckpt & coarse_mesh to logspython main.py --config configs/image.yaml input=data/name_rgba.png save_path=name# gui mode (supports visualizing training)python main.py --config configs/image.yaml input=data/name_rgba.png save_path=name gui=True# load and visualize a saved ckptpython main.py --config configs/image.yaml load=logs/name_model.ply gui=True# use an estimated elevation angle if image is not front-view (e.g., common looking-down image can use -30)python main.py --config configs/image.yaml input=data/name_rgba.png save_path=name elevation=-30### training mesh stage# auto load coarse_mesh and refine 50 iters (~1min), export fine_mesh to logspython main2.py --config configs/image.yaml input=data/name_rgba.png save_path=name# specify coarse mesh path explicitypython main2.py --config configs/image.yaml input=data/name_rgba.png save_path=name mesh=logs/name_mesh.obj# gui modepython main2.py --config configs/image.yaml input=data/name_rgba.png save_path=name gui=True# export glb instead of objpython main2.py --config configs/image.yaml input=data/name_rgba.png save_path=name mesh_format=glb### visualization# gui for visualizing mesh# `kire` is short for `python -m kiui.render`kire logs/name.obj# save 360 degree video of mesh (can run without gui)kire logs/name.obj --save_video name.mp4 --wogui# save 8 view images of mesh (can run without gui)kire logs/name.obj --save images/name/ --wogui### evaluation of CLIP-similaritypython -m kiui.cli.clip_sim data/name_rgba.png logs/name.objPlease check ./configs/image.yaml for more options.
Image-to-3D (stable-zero123):
### training gaussian stagepython main.py --config configs/image_sai.yaml input=data/name_rgba.png save_path=name### training mesh stagepython main2.py --config configs/image_sai.yaml input=data/name_rgba.png save_path=nameText-to-3D:
### training gaussian stagepython main.py --config configs/text.yaml prompt="a photo of an icecream" save_path=icecream### training mesh stagepython main2.py --config configs/text.yaml prompt="a photo of an icecream" save_path=icecreamPlease check ./configs/text.yaml for more options.
Text-to-3D (MVDream):
### training gaussian stagepython main.py --config configs/text_mv.yaml prompt="a plush toy of a corgi nurse" save_path=corgi_nurse### training mesh stagepython main2.py --config configs/text_mv.yaml prompt="a plush toy of a corgi nurse" save_path=corgi_nursePlease check ./configs/text_mv.yaml for more options.
Image+Text-to-3D (ImageDream):
### training gaussian stagepython main.py --config configs/imagedream.yaml input=data/ghost_rgba.png prompt="a ghost eating hamburger" save_path=ghost### training mesh stagepython main2.py --config configs/imagedream.yaml input=data/ghost_rgba.png prompt="a ghost eating hamburger" save_path=ghostHelper scripts:
# run all image samples (*_rgba.png) in ./datapython scripts/runall.py --dir ./data --gpu 0# run all text samples (hardcoded in runall_sd.py)python scripts/runall_sd.py --gpu 0# export all ./logs/*.obj to mp4 in ./videospython scripts/convert_obj_to_video.py --dir ./logsGradio Demo:
python gradio_app.pyTipsThe world & camera coordinate system is the same as OpenGL:WorldCamera+y up target|| / || / |______+x|/______right // //// +zforwardelevation: in (-90, 90), from +y to -y is (-90, 90)azimuth: in (-180, 180), from +z to +x is (0, 90)Trouble shooting OpenGL errors (e.g., [F glutil.cpp:338] eglInitialize() failed):# either try to install OpenGL correctly (usually installed with the Nvidia driver), or use force_cuda_rast:python main.py --config configs/image_sai.yaml input=data/name_rgba.png save_path=name force_cuda_rast=Truekire mesh.obj --force_cuda_rastAcknowledgementThis work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!
gaussian-splatting and diff-gaussian-rasterizationthreestudionvdiffrastdearpyguiCitation@article{tang2023dreamgaussian, title={DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation}, author={Tang, Jiaxiang and Ren, Jiawei and Zhou, Hang and Liu, Ziwei and Zeng, Gang}, journal={arXiv preprint arXiv:2309.16653}, year={2023}}